Search for: All records

We focus on addressing the object counting limitations of vision-language models, with a particular emphasis on Contrastive Language-Image Pre-training (CLIP) models. Centered on our hypothesis that counting knowledge can be abstracted into linear vectors within the text embedding space, we develop a parameter-efficient fine-tuning method and several zero-shot methods to improve CLIP's counting accuracy. Through comprehensive experiments, we demonstrate that our learning-based method not only outperforms full-model fine-tuning in counting accuracy but also retains the broad capabilities of pre-trained CLIP models. Our zero-shot text embedding editing techniques are also effective in situations where training data is scarce, and can be extended to improve Stable Diffusion's ability to generate images with precise object counts. We also contribute two specialized datasets to train and evaluate CLIP’s counting capabilities. Our code is available at https://github.com/UW-Madison-Lee-Lab/CLIP_Counting. 

The evolution from Large Language Models (LLMs) to Multimodal Large Language Models (MLLMs) has spurred research into extending In-Context Learning (ICL) to its multimodal counterpart. Existing such studies have primarily concentrated on image-to-text ICL. However, the Text-to-Image ICL (T2I-ICL), with its unique characteristics and potential applications, remains underexplored. To address this gap, we formally define the task of T2I-ICL and present CoBSAT, the first T2I-ICL benchmark dataset, encompassing ten tasks. Utilizing our dataset to benchmark six state-of-the-art MLLMs, we uncover considerable difficulties MLLMs encounter in solving T2I-ICL. We identify the primary challenges as the inherent complexity of multimodality and image generation, and show that strategies such as fine-tuning and Chain-of-Thought prompting help to mitigate these difficulties, leading to notable improvements in performance. Our code and dataset are available at https://github.com/UW-Madison-Lee-Lab/CoBSAT. 

Studies of moiré systems have explained the effect of superlattice modulations on their properties, demonstrating new correlated phases. However, most experimental studies have focused on a few layers in two-dimensional systems. Extending twistronics to three dimensions, in which the twist extends into the third dimension, remains underexplored because of the challenges associated with the manual stacking of layers. Here we study three-dimensional twistronics using a self-assembled twisted spiral superlattice of multilayered WS2. Our findings show an opto-twistronic Hall effect driven by structural chirality and coherence length, modulated by the moiré potential of the spiral superlattice. This is an experimental manifestation of the noncommutative geometry of the system. We observe enhanced light–matter interactions and an altered dependence of the Hall coefficient on photon momentum. Our model suggests contributions from higher-order quantum geometric quantities to this observation, providing opportunities for designing quantum-materials-based optoelectronic lattices with large nonlinearities. 

Magnetic properties such as coercivity, remanence and saturation magnetization will determine the area enclosed by the hysteresis loop of a magnetic material, which also represents magnetic heating. Nanowarming of cryopreserved organs is a new application for magnetic heating using nanoparticles. In this paper, isolated Ni MNW of different sizes and shapes are studied via micromagnetic simulation to explore the optimization of heating using individual MNW. Ellipsoidal MNWs with small (30nm) diameters turn out to be most promising in heating ability due to their large hysteresis area and their potential to distribute uniformly in an organ that is being heated. In addition to optimized heating, a special switching pattern of magnetic moment was also observed for cylindrical large (200nm) MNW. This special switching pattern can trigger applications such as quantum computing.